Text Mining

In order to mine the data in this way, we used Python over the speeches and words. For each speech, we iterated over all words, and for each relevant word, we collected relevant nearby words (those relevant words within d of the word being considered). Some sample code is provided:

output = [] # each element is one line of the csv output
output.append("word1,word2,weight") # csv header
output_dictionary = defaultdict(lambda: 0) # maps frozenset (of two strings) to int

for input_file in input_files: # these are the speeches
  with open(input_file) as f:
    text = f.read()
    
  regex = re.compile('[^a-zA-Z\' ]')
  text = regex.sub('', text)

  words = text.split(" ")
  for i in range(len(words)):
    word = words[i].lower()
    if word not in top_words: # top_words is the set of relevant words to be considered
      continue
          
    # acquire nearby words
    nearby_words = []
    for j in range(i - dist, i + dist + 1): # check all j within `dist` of word
      if j >= 0 and j < len(words) and j != i: # if j is a valid index
        if words[j] in top_words and words[j] != word:
          nearby_words.append(words[j])

    # increase weights accordingly
    for nearby_word in nearby_words:
      # create hashable frozenset object
      to_hash = frozenset([word, nearby_word])
      output_dictionary[to_hash] += 1

  for pair_set in output_dictionary.keys():
    pair_list = list(pair_set)
    weight = output_dictionary[pair_set]
    output.append(pair_list[0] + ',' + pair_list[1] + ',' + str(weight))

  output = '\n'.join(output)

Note that we actually double count connections between words, but it doesn’t matter for our purposes because we double count all connections, so in comparing these relationships there is no observable difference.

Network Analysis for Popular Election Topics:

Once the data was extracted, we used R’s igraph and ggnetwork to display the networks. Our first set of networks focuses on references to popular election topics we found on usnews.com. We mined the text as described above, and then displayed our results in R. We provide code for our first network:

biden_network_data <- read_csv("biden_speeches_network_data.csv") %>%
  arrange(desc(weight)) %>%
  top_n(16)

biden_network <- graph_from_data_frame(biden_network_data, directed = FALSE)

ggplot(data = ggnetwork(biden_network)
       , aes(x = x, y = y, xend = xend, yend = yend)) +
  geom_edges(arrow=arrow(type="closed", length=unit(6,"pt"))
            , color = "lightgray", aes(size = weight)) +
  geom_nodes() +
  geom_nodelabel(aes(label = name)) +
  theme_blank() +
  theme(legend.position = "none") +
  ggtitle("Topic Relationships in Biden's Speeches")

One interesting takeaway from these visualizations is the more focused quality of Biden’s speeches. While Trump frequently references guns, trade, and Iran in the context of other topics, Biden does not do this as much. A second observation is that both candidates connect similar main topics, including heath care, education, COVID-19, and the economy. For Biden, climate change is one of these topics. (This is not the case for Trump).

Network Analysis for Most Commonly Used Words

Our second set of networks focuses on the most commonly used words across all speeches. The same procedure was used as above.

One observation from these networks is how central the word ‘people’ is to both candidates’ speeches. This is the most commonly used word by both candidates, and therefore it makes sense for it to be central in these networks. Furthermore, Biden’s use of commmon words is more focused, with nearly all of the strongest edges containing the word ‘people’. Trump’s use of these words, on the other hand, is more spread out.